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ABSTRACT 

Faceted  simulator  displays  are  widely  used  because  they  are  relatively  compact  and  economical.  One  drawback, 
however,  is  that  viewing  distance  changes  depending  on  where  users  are  looking.  This  variation  creates  a  challenge 
for  the  integration  of  binocular  head  mounted  displays  (HMDs),  because  confusing  imagery  and  visual  fatigue  can 
result  when  the  user  views  symbology  presented  by  the  HMD  at  one  distance  and  simulator  imagery  at  different 
distances.  Understanding  the  best  approach  to  presenting  symbology  with  a  binocular  HMD  in  a  faceted  simulator 
has  become  an  important  issue,  with  the  deployment  of  the  F-35  Joint  Strike  Fighter  and  its  binocular  HMD. 
Successful  integration  of  a  binocular  HMD  would  not  only  allow  current  faceted  simulators  to  be  retrofit  with  the  F- 
35  simulator  HMD,  but  would  also  allow  future  simulators  to  have  either  a  dome  or  faceted  design,  thus  affording 
acquisition  agencies  greater  flexibility.  Binocular  HMDs  are  becoming  more  prevalent,  so  solving  this  integration 
issue  will  likely  become  important  for  multiple  platforms. 

We  performed  an  experiment  to  quantify  the  best  method  of  presenting  symbology  on  a  binocular  HMD  when  used 
with  a  faceted  simulator  display.  Five  viewing  conditions  were  tested;  1)  HMD  converged  to  36”,  2)  HMD 
converged  to  42”,  3)  dynamic  HMD  vergence,  4)  monocular  presentation  on  the  HMD,  and  5)  on-screen 
presentation.  Screen  distances  ranging  from  36”  to  54”  were  tested. 

Our  results  suggest  that  adaptive  vergence  is  the  preferred  solution.  Both  static  vergence  conditions  and  the 
monocular  condition  resulted  in  lower  comfort  scores  and  poorer  performance.  The  on-screen  condition,  although 
rated  comfortable,  does  not  represent  the  real-world  flight  condition  where  symbology  is  displayed  using  an  HMD. 
Although  additional  evaluations  under  more  operational  conditions  remain  to  be  completed,  these  results  indicate 
that  adaptive  vergence  is  a  viable  solution  for  the  integration  of  binocular  HMDs  into  faceted  flight  simulator 
displays. 
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INTRODUCTION 

Our  previous  work  {Browne,  Moffitt,  &  Winterbottom 
200S)  investigated  visual  anomalies  while  using  a 
binocular  head  mounted  display  (HMD)  in  a  faceted 
flight  simulator  (a  simulator  with  multiple  flat  “tiles” 
instead  of  a  dome).  These  anomalies  included  subject 
reports  of  floating,  buried  or  confusing  symbology, 
doubling  of  symbology  or  the  background  imagery, 
symbology  slanted  relative  to  the  simulator  screen,  and 
general  viewing  discomfort.  Our  primary  conclusion 
was  that  these  visual  anomalies  could  be  problematic 
when  integrating  binocular  HMDs  into  faceted  flight 
simulators. 

The  viewing  discomfort  subjects  reported  was  likely 
caused  by  diplopia  -  or  double  imaging.  Diplopia 
occurs  often  in  the  real  world  when  we  look  at  multiple 
objects  located  at  different  viewing  distances.  When 
directing  our  attention  to  a  given  object,  we 
subconsciously  suppress  double  vision  of  objects  at 
other  distances.  Diplopia  is  problematic  when  using  a 
binocular  HMD  in  a  faceted  simulator  because  there 
are  some  circumstances,  such  as  targeting  an  enemy 
aircraft,  where  the  user  must  view  both  the  out  the 
window  display  (target)  and  the  ITMD  image  (targeting 
reticle)  simultaneously,  as  shown  in  Figure  1. 

Vergence  angle  is  a  strong  depth  cue  -  an  object  is 
seen  as  closer  in  depth  when  the  eyes  are  converged 
(angled  inward)  and  more  distant  when  the  eyes  are 
diverged  (becoming  more  parallel).  Figure  l-left 
shows  the  position  of  the  two  eyes  when  verged  for  the 
out  the  window  (OTW)  image.  If  the  HMD 
symbology  is  in  front  of  the  OTW  image,  as  could 
occur  in  a  faceted  display,  then  that  image  falls  on  non¬ 
matching  locations  in  the  two  eyes,  creating  a  double 
image  of  the  HMD  symbology.  Alternatively,  if  the 
user  shifts  their  vergence  to  the  HMD  symbology’s 
depth,  then  the  OTW  image  will  appear  doubled 
(Figure  1 -right). 


Figure  1.  Graphical  Depiction  Of  Diplopia  Caused 
By  Concentrating  On  Two  Objects  At  Different 
Distances  Simultaneously.  Concentrating  On  The 
OTW  Scene  (Shown  To  The  Left)  Causes  A 
Diplopic  Image  Of  The  Reticle.  Concentrating  On 
The  Reticle  Causes  A  Diplopic  Image  Of  The  OTW 
Scene. 

The  specific  flight  simulator  display  design  of  interest 
for  this  investigation  is  the  M2DART  (Mobile  Modular 
Display  for  Advanced  Research  and  Training),  a 
simulator  display  system  that  can  be  reconfigured  and 
deployed  for  a  variety  of  training  tasks  around  the 
globe  yet  still  provide  panoramic  imagety'  (Wight,  Best 
&  Peppier,  1998).  The  key  to  this  design  is  the  use  of 
a  faceted  display  and  rear  projection.  These  facets 
range  in  viewing  distance  from  as  close  as  36  inches  to 
a  practical  maximum  of  54  inches.  This  range  of 
viewing  distances  creates  the  potential  for  vergence 
mismatch  central  to  our  investigations. 

Figure  2  demonstrates  a  simplified  overhead  view  of  a 
faceted  simulator  like  the  M2DART.  The  green  line 
represents  the  FIMD  symbology,  presented  at  a  fixed 
vergence  distance.  Shown  also  are  two  OTW  display 
facets  (one  straight  ahead,  one  angled  on  the  right).  As 
the  user  looks  at  different  locations  within  the  faceted 
simulator,  the  difference  in  distance  between  the  HMD 
symbology  (green  line)  and  the  display  facet  changes. 
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When  the  user  is  looking  straight  ahead  (Figure  2-left), 
there  is  no  distance  difference.  When  the  user  looks  at 
the  seam  between  the  display  facets  (Figure  2-center) 
there  is  a  significant  distance  difference  at  the  center  of 
the  field  of  view.  When  the  user  looks  at  the  right 
facet  (Figure  2-right)  there  is  a  constantly  varying 
distance  difference.  If  these  distance  differences  get 
too  large,  the  user  will  report  visual  anomalies, 
including  eyestrain,  confusing  imagery  and  diplopia. 


Figure  2.  Simplified  Overhead  View'  Showing  The 
Distance  Difference  Between  The  HMD  Image 
Plane  And  Two  Adjoining  Screens  Of  A  Faceted 
Display  System. 

Since  the  development  of  the  M2DART,  a  variety  of 
manufacturers  have  developed  similar  faceted  display 
designs  (e.g.  Boeing  VIDS,  L.3  SimuSphere,  Glass 
Mountain  Optics  WASP).  These  display  systems  are 
used  widely  across  the  services  for  a  variety  of  training 
applications.  As  such,  we  expect  that  as  long  as  there 
are  faceted  simulators,  the  potential  exists  for  vergence 
mismatch  when  using  a  binocular  FLMD. 

In  air  aircraft,  all  of  the  information  that  the  pilot  views 
outside  the  cockpit  is  at  or  near  optical  infinity 
(distances  from  30  feet  and  beyond).  No  matter  where 
the  pilot  looks,  the  vergence  is  basically  the  same. 
Therefore,  the  HMD  can  be  designed  such  that  for 
aircraft  use,  its  symbology  is  also  converged  to  optical 
infinity.  Because  of  this,  there  is  no  vergence 
mismatch  between  the  OTW  scene  and  the  HMD 
symbology  for  an  HMD  used  in  an  aircraft.  The 
monochrome  HMD  symbology  in  an  aircraft  will 
appear  distinct  from  the  real  world,  and  may  appear 
closer  in  distance,  but  should  not  appear  blurry, 
doubled  or  slanted  relative  to  the  OTW  scene. 

Despite  the  optical  equivalence  of  the  HMD 
symbology  and  real-world  imagery,  there  is  substantial 
evidence  that  the  pilot  switches  attention  between  the 
two  (e.g.,  McCann,  Foyle,  &  Johnston,  1993;  McCann, 
Lynch,  Foyle,  &  Johnston,  1993).  This  attention 
switching  is  not  driven  by  binocular  disparity  and 
focus,  but  by  dissimilar'  visual  imagery  and 
information.  We  postulated  that  increased  visual 
disparity  will  lead  to  an  increase  in  attention  switching 
(and  an  increase  in  time  to  perform  a  task)  and 


designed  the  performance  experiment  to  test  this 
hypothesis. 

Future  military  aircraft  are  likely  to  have  an  HMD  that 
supports  visual  cueing  with  symbology  and  imagery. 
The  Joint  Strike  Fighter  (JSF)  will  be  the  first  US 
fighter  jet  with  a  binocular  HMD  that  serves  as  a 
primary  flight  instrument.  Although  a  dome  display 
may  be  a  good  solution  for  the  F-35  and  other 
platforms,  a  number  of  concerns  have  been  raised 
about  using  dome  simulators.  The  first  is  that  if  all 
faceted  simulators  have  to  be  replaced  with  dome 
simulators,  it  will  be  at  a  large  cost.  The  second 
concern  is  the  size  of  the  training  system  footprint  and 
visual  system  complexity,  which  drive  cost  factors 
associated  with  training  device  installation  and 
sustainment.  These  issues  make  the  integration  of  a 
binocular  HMD  into  smaller,  more  cost-effective 
faceted  displays  of  particular  interest. 

The  above  concerns  make  the  compatibility  of  faceted 
simulators  with  binocular  HMDs  an  urgent  problem  in 
training  F-35  pilots.  The  goal  of  the  present 
experiment  was  to  quantify  vergence  mismatch  effects 
identified  in  our  first  experiment  (Browne,  Moffitt  and 
Winterbottom,  2008).  We  also  wanted  to  identify  the 
best  solutions  for  improving  symbology  appearance, 
viewing  comfort  and  pilot  performance  for  a  binocular 
HMD  integrated  with  a  faceted  display. 

In  our  previous  work,  we  tried  a  number  of  HMD 
viewing  conditions  but  no  single  HMD  configuration 
prevented  all  the  problems  associated  with  integrating 
a  binocular  HMD  with  a  faceted  display.  Binocular 
symbology  converged  to  one  simulator  viewing 
distance  appeared  diplopic  or  doubled  at  other 
distances.  Monocular  symbology  avoided  this 
vergence  mismatch,  but  was  rated  by  most  observers  as 
less  comfortable  to  view  compared  to  binocular 
symbology.  Both  binocular  and  monocular  symbology 
appeared  slanted  relative  to  the  simulator  screen 
whenever  the  symbology  was  not  viewed  straight¬ 
ahead.  On-screen  symbology  exhibited  no  binocular 
problems  and  was  rated  as  comfortable  to  view,  but 
this  solution  does  not  utilize  an  HMD  and  creates  a 
training  situation  that  is  not  representative  of  what 
users  will  see  in  flight. 

In  the  virtual  world  of  HMD  symbology,  vergence  can 
be  manipulated  electronically  by  adjusting  the 
horizontal  binocular  disparity.  Electronically  shifting 
the  left-  and  right-eye  symbology  creates  the 
perception  of  a  change  in  depth.  An  inward  shift 
brings  the  symbology  forward,  while  an  outward  shift 
pushes  the  symbology  away.  We  used  this  concept  to 
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design  an  adaptive  viewing  condition  that 
electronically  and  automatically  adjusts  the  vergence  of 
the  HMD  depending  on  where  the  user  is  looking 
within  the  faceted  display.  One  of  the  goals  of  our 
current  experiment  was  to  validate  this  approach  both 
in  terms  of  comfort  and  performance. 


METHODS 


Observers 

Thirteen  observers  (twelve  male  and  one  female),  ages 
25  to  63,  participated  in  our  experiments.  All  had 
experience  with  the  design,  marketing  or  use  of  HMDs. 
Four  participants  described  themselves  as  “very 
familiar”  with  HMDs.  Four  subjects  were  fighter 
pilots,  three  of  whom  had  extensive  experience  with 
the  monocular  joint  helmet  mounted  cueing  system 
(JHMCS)  HMD.  The  interpupillary  distance  (IPD)  of 
each  subject  was  measured  with  an  L8  pupilometer, 
model  NH-L8.  Eye  dominance  was  determined  by 
noting  which  eye  was  used  for  sighting  through  an 
aperture.  Eight  participants  were  right-eye  dominant 
and  five  were  left-eye  dominant.  Following  helmet 
fitting  and  HMD  adjustment,  the  resting  position  of 
vergence  was  measured  using  nonius  targets  with  a 
dark  background.  Intact  stereo  vision  was  ascertained 
by  presenting  three  depth  planes  of  symbols,  and 
asking  the  subject  to  identify  the  close,  intermediate 
and  distant  figures.  Finally,  a  binocular  acuity  level  of 
2.6  minutes  or  20/52  was  verified  with  on-screen 
“tumbling  Es”  presented  at  the  straight-ahead  viewing 
distance  of  36  inches.  Six  of  the  participants  wore 
eyeglasses,  three  of  whom  had  progressive  lenses. 

Stimuli  and  Apparatus 

A  Rockwell-Collins  Optronics  SimEye  SXL50  STM 
binocular  see-through  HMD  was  used  for  our 
experiment.  This  HMD,  shown  in  Figure  3,  was 
designed  as  a  simulator  HMD  for  the  F-35  Joint  Strike 
Fighter.  It  has  a  1280  x  1024  pixel  format,  40  x  30 
degree  field-of-view  (FOV),  and  monochrome  green 
imagery.  Focus  of  this  unit  was  0.9  diopter  (not 
accounting  for  chromatic  aberration)  with  a  nominal 
optical  vergence  distance  of  34.5”  (88  cm).  This  HMD 
has  a  see-through  transmission  of  >70%  and  was  set  to 
a  nominal  luminance  of  6.5  fL.  This  I-IMD  clasps  onto 
an  HGII-55/P  military  flight  helmet.  Three  helmet 
sizes  were  available  for  this  study:  M,  L  &  XL. 


Figure  3.  SimEye  SXL50  HMD. 


A  Sony  SXRD  3-panel  LCDS  (liquid  crystal  on 
silicon)  1920  x  1080  pixel  monitor  with  a  nominal 
luminance  of  35  fL  (at  the  brightest  portion  of  the  sky) 
was  used  to  present  the  out-the-window  (OTW) 
imagery.  The  Sony  monitor  had  a  horizontal  dimension 
of  52”,  as  shown  in  Figure  4.  The  straight  ahead 
viewing  distance  was  36”  and  the  nominal  viewing 
position  was  9”  in  from  the  left  edge  of  the  screen. 
This  allowed  us  to  test  viewing  distances  ranging  from 
36”  (straight  ahead)  to  54”  (near  the  upper  right  hand 
comer).  The.se  distances  encompassed  the  minimum 
and  maximum  viewing  distances  of  the  M2DART. 


36" 


(near  upper 
right  comer) 


Figure  4:  Layout  of  Simulator  Screen 
and  Subject  Positioning 

The  trade-off  involved  with  replicating  the  M2DART 
viewing  conditions  on  a  single  monitor  was  that  our 
experiment  presented  symbology  at  a  more  severe 
apparent  slant  angle  than  found  in  the  M2DART. 
Some  observers  commented  on  this  slant  but  we  think 
that  overall  the  conclusions  for  our  experiment  transfer 
to  the  M2DART. 

A  PC  computer  with  nVidia  GeForce  graphics  cards 
provided  the  imagery  for  both  the  HMD  and  monitor. 
A  Polhemus  Liberty  tracker  transmitted  head  position 
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and  angle  relative  to  the  OTW  display.  Testing  took 
place  in  an  area  surrounded  by  black  curtains  with  a 
nominal  illuminance  of  1.5  fc.  HMD  luminance, 
monitor  luminance  and  room  illuminance  were 
periodically  tested  to  ensure  nominal  levels  were 
maintained. 

The  experimental  tasks  used  an  HMD  reticle  and  an 
OTW  image  of  a  desert  scene  including  sky, 
mountains,  and  a  distant  city.  The  HMD  reticle,  shown 
in  Figure  5,  subtended  a  total  of  8  x  6  degrees,  with  the 
center  circle  subtending  1.5  degrees.  Each  target  on 
the  OTW  display  was  a  51  x  22  pixel  helicopter  image 
subtending  2.1  x  0.9°  degrees  at  the  straight-ahead  36” 
screen  position.  The  helicopter  was  gray  with  a  green 
center  dot  to  aid  in  centering  the  reticle  on  it.  A  photo 
of  a  person  wearing  the  HMD  and  viewing  the  monitor 
is  shown  in  Figure  6. 


Figure  6.  Subject  Looking  For  A  Target 


Procedure 

We  asked  subjects  to  complete  four  activities  as  part  of 
our  experiment;  boresighting,  reticle  distance 
preference,  viewing  comfort  and  visual  search  and 


performance.  Each  of  these  activities  are  discussed 
further  in  the  next  sections. 

Boresighting 

To  calibrate  the  head-tracker  prior  to  each  series  of 
trials,  the  system  was  boresighted  to  the  relevant  target 
locations.  To  do  this,  a  target  was  presented  on  the 
OTW  display  and  the  subject  aligned  the  HMD  reticle 
with  the  target  and  pressed  a  mouse  button. 

Reticle  Distance  Preference 

The  apparent  viewing  distance  of  the  HMD  reticle  can 
be  changed  by  electronically  shifting  the  images  on  the 
left  and  right  eye  microdisplays  inside  the  HMD.  We 
measured  the  preferred  reticle  distance  for  each  of  ten 
screen  targets  located  along  horizontal  lines  that 
divided  the  display  vertically  into  quarters,  and  at 
distances  of  36”  to  54”  in  2”  increments.  The  primary 
purpose  of  this  exercise  was  to  establish  preferred 
positions  for  the  FIMD  vergence  as  a  function  of 
viewing  distance.  We  used  this  information  to  set 
vergence  dynamically  for  the  adaptive  viewing 
condition  in  the  viewing  comfort  and  performance 
experiments. 

Participants  began  with  the  straight-ahead  36”  target 
and  adjusted  the  apparent  viewing  distance  of  the 
HMD  reticle  inward  and  outward  using  the  left  and 
right  buttons  of  a  mouse.  They  were  instructed  to  put 
the  reticle  at  the  position  that  they  felt  would  work  best 
for  them  to  complete  a  targeting  task,  not  necessarily  at 
the  same  distance  as  the  screen.  Target  presentation 
progressed  from  left  to  right  until  the  final  54”  target 
was  completed.  For  each  target,  head-tracker 
infonnation  was  recorded  with  the  reticle  distance 
(stored  as  the  lateral  shift  of  the  left-  and  right-image 
on  the  HMD  display).  Figure  7  shows  approximately 
how  the  HMD  imagery  appeared  relative  to  the  OTW 
image  for  near,  approximately  equal,  and  more  distant 
vergence  settings. 


Figure  7.  Appearance  Of  HMD  Symbology 
Relative  To  OTW  Imagery  As  The  Viewer 
Electronically  Adjusted  Vergence  Settings  On  The 
HMD 
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Viewing  Comfort  Rating 

We  expanded  on  our  initial  work  to  check  viewing 
comfort  as  a  function  of  viewing  condition  (Browne  et 
al.).  We  changed  the  experiment  to  include  better 
control  over  the  reticle  distance  independent  of  subject 
interpupilliary  distance.  We  also  checked  viewing 
comfort  over  more  distances  and  included  an  adaptive 
condition  along  with  the  fixed  binocular,  monocular 
and  on  screen  conditions.  For  this  adaptive  condition, 
we  used  the  head  tracker  to  indicate  where  the  subject 
was  looking  and  changed  the  vergence  such  that  the 
reticle  viewing  distance  was  approximately  the  same  as 
the  distance  from  the  user  to  the  OTW  display. 

Each  trial  started  with  practice  targets  located  at  three 
screen  positions.  Subjects  were  instructed  to  first 
preview  the  task  by  rotating  their  head  to  position  the 
reticle  over  each  target  to  see  what  different  vergence 
mismatches  looked  like. 

Once  the  practice  session  was  over,  the  subject 
returned  to  the  starting  position,  positioned  the  reticle 
over  the  36”  straight-ahead  target  and  called  out  a 
rating  of  viewing  comfort:  “Not  uncomfortable  to 
view”,  "Somewhat  uncomfortable  to  view”  or  “Very 
uncomfortable  to  view”.  This  sequence  was  repeated 
for  seven  targets  located  from  36  to  54  inches  along  the 
horizontal  mid-line  in  3”  increments. 

The  not/somewhahvery  uncomfortable  ratings  were 
adapted  from  similar  scales  with  approximately  equal 
intervals  (e.g.,  Babbitt  &  Nystrom,  1989).  Subjects 
were  instructed  that  “Not  uncomfortable  to  view” 
means  that  the  image  may  look  unusual,  but  not  blurry, 
doubled  or  confusing.  You  could  easily  view  this 
image  for  a  period  of  time.  “Very  uncomfortable  to 
view”  means  the  image  may  be  blurry,  doubled  or 
confusing — something  you  would  not  want  to  view  for 
any  length  of  time.  “Somewhat  uncomfortable  to 
view”  describes  a  viewing  comfort  between  these  two 
e.xtremes. 

Five  viewing  conditions  were  tested:  1)  Vergence  set 
to  the  preferred  reticle  distance  for  the  36”  target,  2) 
Vergence  set  to  the  preferred  reticle  distance  for  the 
42”  target,  3)  An  adaptive  distance  that  adjusted  the 
reticle  to  the  user  preferred  distance  for  each  target,  4) 
Monocular,  presented  in  the  right-eye,  and  5)  On- 
Screen,  where  the  reticle  was  drawn  on  the  OTW 
display. 

Visual  Search  and  Performance 

In  this  experiment  we  investigated  whether  the  viewing 

condition  is  simply  a  comfort  and  perception  issue,  or 


if  visual  performance  was  also  affected.  Participants 
were  asked  to  search  for  helicopter  targets  located  at 
ten  locations  on  the  screen.  These  locations  ranged  in 
distance  from  36”  to  54”  in  2”  increments.  The 
helicopter  could  appear  in  any  of  10  positions,  but  all 
positions  were  randomly  presented  three  times  during 
the  course  of  the  experiment.  When  a  target  was 
located,  the  subject  aligned  the  HMD  reticle  with  the 
target  and  the  reticle  changed  to  a  “missile  lock” 
configuration  (four  aiTOWs  suiTOunding  the  targeting 
reticle,  as  shown  in  Figure  8).  After  one  second  of 
accurate  alignment  (representing  a  “missile  lock”),  a 
three-digit  number  was  displayed  on  the  reticle  and 
another  three-digit  number  was  displayed  on-screen. 
For  the  on-screen  viewing  condition,  both  numbers 
were  displayed  on-screen.  These  spatially  adjacent 
numbers  were  compared  by  the  participant.  The 
participant  was  instructed  to  push  the  left  mouse  button 
if  they  were  the  same,  and  the  right  button  if  different. 
If  an  error  was  made,  the  subject  was  required  to  enter 
the  correct  response.  After  a  correct  response,  a  new 
target  randomly  appeared  at  one  of  the  remaining 
locations,  and  the  subject  continued  the  search  task. 
The  numbers  were  displayed  against  a  dark 
background  to  ensure  consistent  contrast  across  all 
target  locations. 


The  five  viewing  conditions  tested  in  the  viewing 
comfort  experiment  were  also  used  for  this  experiment. 
The  preference  data  from  the  first  experiment  were 
used  to  adjust  reticle  distance  for  the  adaptive 
condition.  Three  sequential  trials  were  run  for  each 
condition.  We  randomized  the  order  of  presentation  of 
each  viewing  condition  across  subjects  to  minimize 
learning  and  fatigue  effects  on  the  aggregate  data. 

We  designed  this  experiment  to  replicate  conditions  in 
a  real  simulator  environment  where  operators  must 
redirect  their  attention  between  HMD  symbology  and 
on-screen  imagery.  The  primary  data  from  this  task 
were  total  time  to  find  and  compare  numbers  at  the  ten 
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target  locations.  We  also  recorded  single  target  times 
from  acquisition  to  response. 


RESULTS 

Viewing  Comfort  Ratings 

The  results  of  the  viewing  comfort  experiment  are 
shown  in  a  Marimekko  chart  (Figure  9).  We  feel  this 
type  of  chart  provides  a  good  graphical  representation 
of  user  comfort  as  a  function  both  of  viewing  condition 
and  of  viewing  distance.  Each  cell  depicts  the  count  of 
each  rating  choice  for  13  participants.  The  proportion 
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of  each  ceil  which  is  shaded  white  gives  an  indication 
of  the  comfort  of  each  viewing  condition.  For 
example,  all  subjects  rated  the  on-screen  viewing 
condition  at  36”  (located  in  the  lowest,  left-most  cell) 
as  comfortable,  thus  it  is  shaded  completely  white.  On 
the  other  hand,  the  binocular  36”  viewing  condition  at 
a  54”  target  distance  was  rated  very  uncomfortable  by 
almost  every  subject,  so  that  its  cell  (upper  right-most 
cel!)  is  shaded  almost  completely  black. 


Figure  9.  Comfort  Ratings  For  The  Five  Viewing  Conditions  At  Seven  Target  Distances 


The  rating  of  viewing  comfort  at  each  on-screen  target 
distance  was  dependent  on  the  viewing  condition.  The 
binocular  36”  reticle  shows  a  striking  decrease  in 
comfort  as  the  target  distance  moves  beyond  36”.  For 
the  42”  reticle,  the  most  comfortable  viewing  is  found 
at  42”  and  45”,  while  nearer  and  further  target 
distances  resulted  in  greater  reports  of  discomfort. 
These  results  indicate  that  a  fixed-distance  HMD 
reticle  only  results  in  comfortable  viewing  at  or  near  a 
corresponding  target  distance,  in  general  agreement 
with  Browne,  et  al. 


The  binocular  adaptive  viewing  condition  used  the 
preference  data  from  the  first  experiment  to  adjust  the 
FIMD  reticle  distance  based  on  target  distance.  The 
results  in  Figure  9  show  mostly  comfortable  viewing  at 
all  target  distances.  The  few  somewhat  or  very 
uncomfortable  ratings  for  the  more  distant  targets  may 
result  from  the  increase  in  reticle/screen  slant  angle. 
This  angle  may  look  confusing  to  some  participants 
when  presented  in  the  context  of  this  experiment. 
Despite  these  minor  issues,  the  binocular  adaptive 
viewing  condition  was  very  successful  at  mitigating 
vergence  mismatch  compared  to  the  other  HMD-based 
methods. 
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Results  from  Browne,  et  al,  plus  our  previous 
experience  with  monocular  HMDs  indicate  that 
monocular  viewing  is  problematic,  and  the  data  for  the 
monocular  condition  appear  to  confirm  this 
expectation.  The  most  discomfort  was  found  at  the 
closer  target  distances  but  some  subjects  even  found 
the  more  distant  targets  uncomfortable  to  view  with  a 
monocular  presentation. 

The  on-screen  reticle  was  in  perfect  correspondence 
with  the  screen  target  and  imagery  in  terms  of  depth, 
but  it  resulted  in  a  high  degree  of  slant  relative  to  the 
line-of-sight,  and  it  decreased  in  angular  size  with 
distance.  Even  so,  this  viewing  condition  was  rated  as 
most  comfortable  to  view,  in  agreement  with  Browne, 
et  al.  (2008). 

Visual  Search  and  Performance 

We  calculated  the  median  search  time  (time  to  find  the 
target  and  achieve  “lock”),  the  median  response  time 
(time  between  “lock”  and  a  correct  response  on  the 
number  matching  task)  and  the  total  time  (sum  of 
search  and  response  times).  The  average  search  time 
for  all  ten  targets  ranged  from  38  to  41  seconds,  and 
the  effect  of  viewing  condition  was  not  significant. 
Although  we  measured  large  differences  in  the  ratings 
of  comfort  for  the  five  viewing  conditions,  it  did  not 
translate  into  a  difference  in  overall  performance  or  in 
search  time.  Both  of  these  metrics  include  a 
disproportionate  amount  of  time  searching  for  the  next 
target.  The  amount  of  time  spent  on  searching  for  a 
target  might  be  too  coarse  of  a  measure  to  identify 
differences  in  performance  among  viewing  conditions. 

In  contrast,  response  time  was  directly  related  to  the 
reticle  viewing  condition.  We  defined  response  time  as 
the  elapsed  time  between  target  acquisition  and  a 
correct  manual  response.  We  feel  that  response  time  is 
a  good  metric  for  proving  our  hypothesis  that  the 
bigger  the  disparity  is  between  HMD  and  OTW 
imagery,  the  longer  it  will  take  the  subject  to  make  a 
correct  response.  While  differences  in  background 
imagery  and  target  location  could  affect  the  time  to 
find  each  target,  the  effect  on  response  time  should  be 
minimal. 

We  found  significant  effects  for  Viewing  Condition 
[F(4,  48)  =  3.69,  p  <  .05];  Tai-get  Position  [F(9,I08)  = 
5.07,  p  <  .001];  and  the  interaction  of  Viewing 
Condition  and  Target  Position  [F(36,432)  =  1.74,  p  < 
.01]. 


The  aggregate  median  response  times  across  viewing 
conditions  are  shown  in  Figure  10.  The  adaptive 
viewing  condition  had  the  quickest  aggregate  median 
response  time  of  1115  msec.  Just  slightly  slower  was 
the  onscreen  condition,  with  a  time  of  1 157  msec.  The 
fixed  vergence  conditions  were  more  than  10%  slower 
than  the  adaptive  or  onscreen  cases  and  the  monocular 
viewing  condition  provided  the  longest  response  times. 

The  data  show  rather  lai'ge  error  bars  (standard 
deviation),  representing  the  significant  subject  to 
subject  variation  for  each  viewing  condition,  but  we 
believe  that  the  trend  is  obvious  with  adaptive  and 
onscreen  taking  the  least  time  and  monocular  taking 
the  most.  This  indicates  that  not  only  are  the  fixed 
vergence  conditions  and  the  monocular  condition  less 
comfortable,  but  they  also  have  an  impact  on  user 
performance,  at  least  for  a  targeting  task. 

Figure  11  shows  how  the  response  time  varies  as  a 
function  not  only  of  viewing  condition  but  also  of 
target  location.  We  split  the  viewing  conditions  into 
two  graphs  so  that  the  data  is  easier  to  visualize.  The 
adaptive  data  appears  on  both  graphs.  Since  it  was  the 
most  likely  condition  to  be  implemented  in  a  real 
system,  we  wanted  to  ensure  that  we  could  compare  all 
other  viewing  conditions  to  it. 

There  is  a  large  variation  in  response  time  across 
viewing  conditions  for  the  very  near  tai’gets,  with 
relatively  rapid  responses  for  some  viewing  conditions 
(on-screen  and  adaptive)  and  others  (binocular  36”  and 
42”)  with  significantly  slower  response  times.  In  the 
mid-distances,  there  were  differences  between  the 
viewing  conditions,  but  no  obvious  trends.  At  the  far 
distances  performance  for  al!  viewing  conditions 
worsened,  while  corresponding  ratings  of  “somewhat” 
and  “very  uncomfortable”  were  only  found  with 
binocular  36”  and  42”  conditions.  We  attribute  this 
increased  response  time  at  the  longest  target  distances 
to  the  fact  that  the  font  size  and  contrast  were  reduced 
when  viewed  from  these  distances.  Numerous  subjects 
stated  that  the  numbers  were  very  hard  to  read  at  the 
longer  target  distances.  We  tried  to  balance  the 
experiment  by  not  making  the  numbers  too  easy  to  read 
at  the  short  distances  so  that  there  would  be  the 
potential  for  a  noticeable  difference  between  different 
viewing  conditions.  We  may  need  to  increase  the 
readability  of  the  compared  numbers  for  future 
experiments. 
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Figure  10.  Average  Median  Response  Time  As  A  Function  Of  Viewing  Condition 
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Figure  11.  Average  Median  Response  Time  As  A  Function  Of  Viewing  Condition  And  Target  Distance 


DISCUSSION 

We  investigated  the  effect  of  five  viewing  conditions 
on  viewing  comfort  and  visual  performance  for  a 
binocular  HMD  integrated  with  a  faceted  OTW 
display.  Our  experiments  confirmed  that  setting  the 
HMD  reticle  to  a  single  viewing  distance  is  not  a  good 
solution  for  the  36”  to  54”  range  of  viewing  distances 
found  in  the  M2DART  faceted  simulator  display.  We 
not  only  found  that  a  static  vergence  viewing  condition 
was  uncomfortable  to  view  over  many  viewing 
distances  but  also  that  a  static  vergence  negatively 
impacted  perfonnance  on  a  targeting  task.  In  addition, 
for  a  fixed  vergence  the  question  always  remains  to 
what  distance  should  the  vergence  be  set?  If  it  is  set  at 
the  straight  ahead  distance  of  36”,  all  otiier  locations 
have  a  vergence  setting  which  is  too  close  and  the 


more  distant  OTW  positions  will  be  uncomfortable  to 
view.  If  the  dioptric  average  distance  of  42”  is  chosen, 
most  users  are  not  satisfied  with  the  viewing  comfort 
straight  ahead  at  36”,  or  at  larger  distances  of  54”. 

Monocular  viewing  was  also  found  to  be 
uncomfortable  to  view,  primarily  at  close  screen 
distances,  but  some  users  found  it  uncomfortable  even 
at  the  longer  target  distances.  These  problems 
extended  to  the  performance  task  as  well,  where  the 
monocular  condition  produced  the  slowest  response 
times  of  all  conditions  tested  at  any  viewing  distance. 

These  problems  with  monocular  imaging  are 
interesting  because  most  current  HMD  systems  used  in 
U.S.  fighter  jets  and  helicopters  are  monocular 
(JHMCS,  DASH,  IHADSS).  In  fact,  a  number  of  the 
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subjects  who  were  pilots  and  had  flown  JHMCS 
expressed  concern  about  the  comfort  of  monocular 
HMD  imageiy  once  they  experienced  binocular  HMD 
imagery.  Whether  or  not  this  monocular  discomfort 
and  reduction  in  performance  can  be  solved  with 
training  is  beyond  the  scope  of  this  experiment,  but  it 
suggests  future  research  into  the  efficacy  of  monocular 
imaging  for  both  simulation  and  flight  hardware. 

The  adaptive  condition  was  more  comfortable  to  view 
compared  to  any  of  the  other  viewing  conditions  with 
the  exception  of  on-screen.  This  condition  also 
showed  the  best  visual  performance  in  the  search  task, 
providing  the  shortest  response  times  of  any  viewing 
condition  tested. 

The  on-screen  condition  was  the  most  comfortable  to 
view  and  resulted  in  good  response  time  performance, 
second  only  to  the  adaptive  condition.  There  are  a 
number  of  issues;  however,  with  using  on-screen 
symbology  to  train  pilots,  including  the  fact  that  the 
on-screen  symbology  follows  the  slant  of  the  simulator 
screen  and  does  not  look  geometrically  correct.  This 
was  especially  true  at  the  farthest  distance  of  54”.  In 
addition,  using  on-screen  symbology  will  have 
significant  visual  differences  to  users  compared  to 
using  symbology  presented  by  an  HMD.  Since  the 
HMD  is  a  primary  flight  instrument  for  the  F-35 
aircraft,  we  believe  that  it  would  be  much  better  to 
train  users  with  imagery  in  the  simulator  that  looks  as  it 
will  in  flight. 

One  additional  concern  is  that  although  vergence  is 
changed  dynamically  with  the  adaptive  viewing 
condition,  we  do  not  change  the  focus  of  the  HMD  as 
the  user  looks  at  different  OTW  distances.  Normally, 
focus  and  vergence  are  coupled,  so  objects  at  the  same 
depth  will  appear  in  focus,  and  objects  at  a  different 
depth  could  appear  blurred.  Although  dynamic 
vergence  would  manipulate  vergence  independently  of 
focus,  which  can  potentially  be  problematic  for  large 
discrepancies  (Hofftnan,  Girshick,  Akeley,  &  Banks, 
2008),  we  believe  the  small  distances  we  are  concerned 
with  will  be  within  a  comfortable  range  and  within  the 
depth  of  focus  of  the  eye  (Winterbottom,  Patterson, 
Pierce,  Covas,  &  Winner,  2007). 

Winterbottom,  et  al  showed  that  depth  of  focus  was  not 
likely  to  be  an  issue  provided  the  difference  in  depth 
between  two  images  was  within  a  reasonable  dioptric 
range,  as  it  is  for  our  experiment  and  in  the  M2DART. 
Hoffman,  et  al  (2007)  showed  that  decoupling  of 
vergence  and  accommodation  can  potentially  create 
discomfort  (at  a  0.67  diopter  (D)  separation).  The 
maximum  separation  of  0.18  D  used  in  our  experiment 


and  in  the  M2DART  was  much  less  than  the  0.67  D 
tested  by  Hoffman,  et  al  and  thus  unlikely  to  cause 
discomfort.  The  maximum  distance  difference  in  the 
M2DART  represents  a  worst  case.  Other  faceted 
simulators,  with  smaller  facets,  would  require  dynamic 
vergence  adjustments  of  even  less  than  the  0.18  D 
range  that  we  tested  over. 

In  summary,  we  believe  that  adaptive  vergence 
provides  a  viable  solution  for  integrating  a  binocular 
HMD  with  faceted  display  systems.  This 
recommendation  is  supported  not  only  by  viewing 
comfort  data,  but  also  by  user  perfonnance  data.  We 
believe  these  results  apply  not  only  to  the  .M2DART, 
but  also  to  other  faceted  simulator  displays  with  similar 
viewing  distance  ranges. 

We  will  seek  to  confirm  our  conclusions  by  integrating 
the  F-35  simulator  HMD  with  dynamic  vergence 
control  into  a  state-of-the-art  faceted  display  system 
with  pilots  executing  realistic  training  tasks  under  more 
operational  conditions. 

Development  of  an  adaptive  vergence  control  system 
would  provide  existing  users  of  faceted  display 
systems  with  a  solution  for  integrating  binocular 
HMDs  for  future  training  applications,  and  provide 
acquisition  agencies  with  additional  alternatives  when 
evaluating  competing  display  system  designs  for 
training  systems  requiring  binocular  HMDs  in 
simulators. 
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